YellowFin: Adaptive optimization for (A)synchronous systems

Hyperparameter tuning is one of the most time-consuming steps in machine learning. Adaptive optimizers, like AdaGrad and Adam, reduce this labor by tuning an individual learning rate for each variable. Lately, researchers have shown interest in simpler methods like momentum SGD as they often yield better results. We ask: can simple adaptive methods based on SGD perform well? We show empirically that hand-tuning a single learning rate and momentum makes SGD competitive with Adam. We analyze momentum’s robustness to learning rate misspecification and curvature variation. We use this robustness to design YellowFin, an automatic tuner for momentum and learning rate in SGD. YellowFin uses a negative-feedback loop to compensate for the added dynamics in asynchronous-parallel settings on the fly. We empirically show YellowFin can converge in fewer iterations than Adam on ResNet and LSTM models, with a speedup of up to 3.28x in synchronous and up to 2.69x in asynchronous settings. ACM Reference Format: Jian Zhang and Ioannis Mitliagkas. . YellowFin: Adaptive optimization for (A)synchronous systems: Extended Abstract. In Proceedings of SysML (SysML’18). ACM, New York, NY, USA, 3 pages.